The present invention relates to a method for identifying colors in an image. The image is in the field of view of a camera or cameras and the camera is an interface to a computer for machine and computer vision applications. The invention also relates to a triggering mechanism in the field of view by identifying colors which allows the camera to be used as an interface with a computer for consumer applications.
Machine vision, commonly called automated inspection, has been used in manufacturing processes to improve productivity and quality. On a typical production line, a sensor detects a part and signals a video camera positioned above or to the side of the inspection point to capture an image and send it to a machine vision processor. Using a combination of machine vision software and hardware, the vision system analyzes the image and provides mathematical answers about the part. A traditional grayscale machine vision technology makes decisions based on 0-256 shades of gray. A typical vision algorithm segments an image into pixels that fall within an intensity band bounded by a lower and an upper threshold from the irrelevant pixels that have intensities outside of this intensity band. Alternatively they look at the rate of change of the image pixels. Once the relevant pixels have been identified, adjacent pixels are clumped together to form blobs or geometric edges and these are then characterized by geometric characteristics such as location, size, shape, etc. Inspecting colored parts or objects with grayscale machine vision system becomes usually unreliable in many cases and impossible in others. For this reason, use of a color machine vision technology is needed to inspect parts or objects in ways that could not be done using traditional grayscale machine vision systems.
Thus far, color machine vision systems have been used for three primary vision applications:
Color Matching—verifying that a certain part's or object's color matches what the vision system is programmed to find
Color Sorting—sorting parts or objects based on color
Color Inspection—inspecting colored parts or objects for defects or imperfections that grayscale image processing tools can't detect.
Defined as the perceptual result of visible light reflected from an object to our eyes, color represents an interpretive concept. Depending on how light is reflected, we all see colors a bit differently. Human visual system uses color to draw conclusions about surfaces, boundaries and location of objects in a scene. It takes three things to see color: a light source, a sample or an object, and a detector such as an eye or a camera. Color derives from the spectrum of light (distribution of light energy versus wavelength) interacting in the eye with the spectral sensitivities of light receptors. Typically, a wavelength spectrum from 380 nm to 740 nm (roughly) of light is detectable by human eye. This range is known as the visible light. The pure “spectral colors” from a continuous spectrum can be divided into distinct colors: violet (˜380-440 nm), blue (˜440-485 nm), cyan (˜485-500 nm), green (˜500-565 nm), yellow (˜565-590 nm), orange (˜590-625 nm), and red (˜625-740 nm). However, these ranges are not fixed, the division is a matter of culture, taste, and language. For instance, Newton added a seventh color, indigo, as wavelengths of 420-440 nm between blue and violet, but most people are not able to distinguish it. Of course, there are many color perceptions that by definition cannot be pure spectral colors. Some examples of non-spectral colors are the “achromatic colors” (black, gray, and white) and colors such as pink, tan, and magenta.
An additive color system involves light “emitted” from a source or illuminant of some sort such as TV or computer monitor. The additive reproduction process usually uses red, green, and blue which are the “primary colors” to produce the other colors. Combining one of these primary colors with another in equal amounts produces the “secondary colors” cyan, magenta, and yellow. Combining all three primary lights (colors) in equal intensities produces white. Varying the luminosity of each light (color) eventually reveals the full gamut of those three lights (colors).
Results obtained when mixing additive colors are often counterintuitive for people accustomed to the more everyday subtractive color system of pigments, dyes, inks, and other substances which present color to the eye by “reflection” rather than emission. Anything that is not additive color is subtractive color.
Light arriving at an opaque surface is either “reflected”, “scattered”, or “absorbed” or some combination of these. Opaque objects that do not reflect specularly (that is, in a manner of a mirror) have their color determined by which wavelengths of light they scatter more and which they scatter less. The light that is not scattered is absorbed. If objects scatter all wavelengths, they appear white. If they absorb all wavelengths, they appear black. Objects that transmit light are either translucent (scattering the transmitted light) or transparent (not scattering the light).
The color of an object is a complex result of its surface properties, its transmission properties, and its emission properties, all of which factors contribute to the mix of wavelengths in the light leaving the surface of an object. The perceived color is then further conditioned by the nature of the ambient illumination, and by the color properties of other objects nearby; and finally, by the permanent and transient characteristics of the perceiving eye and brain.
Light, no matter how complex its composition of wavelengths, is reduced to three color-components by the eye. For each location in the visual field, the three types of color receptor cones in the retina yield three signals based on the extent to which each is stimulated. These values are sometimes called “tristimulus values”.
To analyze and process images in color, machine vision systems typically use data from color spaces such as RGB, HSI (or HSL), HSV (or HSB), CIELAB (or CIEXYZ) CMYK, etc. In the RGB color space, each color appears in its primary spectral components of red, green, and blue. When combined with a three-dimensional coordinate system, the RGB color space defines quantitatively any color on the spectrum. RGB uses “additive” color mixing. X-axis specifies the amount of red color, Y-axis specifies the amount of green and the Z-axis specifies amount of blue. If RGB color model is implemented in 256 (0 to 255) discrete levels of each color component (8 bits) then the color space defines a gamut of 256×256×256 or about 16.7 million colors.
The HSI color space, also known as HSL is broken down into hue, saturation and intensity or lightness. Hue refers to pure color, saturation refers to the degree or color contrast, and intensity refers to color brightness.
HSV (hue, saturation, value), also known as SHB (hue, saturation, brightness), is quite similar to HSL “brightness” replacing “lightness”. Artists often use HSV color space because it is more natural to think about a color in terms of hue and saturation.
CIE 1931 XYZ color space is the first attempt to produce a color space based on measurements of human color perception. It is the most complete color space used conventionally to describe all the colors visible to human eye. It was developed by the “International Commission on Illumination” (CIE). CIE 1976 LAB is based directly on CIE 1931 XYZ color space as an attempt to make the perceptibility of color differences linear. CIE is the most accurate color space but is too complex for everyday uses.
CMYK uses subtractive color mixing in used printing process. It is possible to achieve a large range of colors seen by humans by combining cyan, magenta, and yellow transparent dyes/inks on a white substrate. Often a fourth black is added to improve reproduction of some dark colors. CMYK stores ink values for cyan, magenta, yellow, and black. There are many CMYK color spaces for different sets of inks, substrates, and press characteristics.
Although dozens of defined color spaces exist, color machine vision applications primarily have used RGB and HSI or HSV color spaces.
Prior art systems use various techniques to measure and match colors such as discusses a color sorting method for wires by comparing the output signal of a camera to the intensity ratio of known colors until a substantial match is found.
Another technique provides a color sorting system and method used for sorting fruits and vegetables. The sorting process is handled with a look up table. The pixel value of the input image is sent to the look up table and the output from the look up table is either series of 0's (accept) or 1's (reject).
Another method for automatically and quantitatively measuring color difference between a color distribution of an object and a reference color image uses “color distance” in a color system. A template representing the reference color image is stored in a memory of a machine vision system. The machine vision system generates a sample color image of the object and processes the template together with the sample color image to obtain a total color distance.
An apparatus is known for sorting fragments of titanium-based sponge on the basis color by comparing the color values of the image to a set of data values stored in a look up table for rejection or acceptance of each fragment.
Another system and method for locating regions in a target image by matching a template image with respect to color and pattern information either by using a hill-climbing technique or fuzzy logic.
A different system and method of perceptual color identification can be used for the identification and tracking of objects, for example, in a surveillance video system. The described method includes a multilevel analysis for determining the perceptual color of an object based on observed colors. This multilevel analysis can include a pixel level, a frame level, and/or a sequence level. The determination makes use of color drift matrices and trained functions such as statistical probability functions. The color drift tables and function training are based on training data generated by observing objects of known perceptual color in a variety of circumstances.
It is clear from the prior art that traditional grayscale machine vision systems are being used successfully in a wide variety of inspection and process control applications for the electronic, automotive, food products, packaging, pharmaceutical, and recycling industries. However, the use of color machine vision systems in these industries has only been applicable to well controlled immediate environments or surroundings. It is also clear that prior art relied on matching color to a reference color image or template. A color machine and computer vision system that can make robust identification of color under varying lighting and changing image shift, scale, and rotation conditions is desirable. Machine vision systems use specialized and expensive hardware and software and therefore their use has been limited to industrial applications. With the advance of inexpensive color webcams, it is also desirable to find use for computer vision systems in cost sensitive consumer applications.
It would desirable to provide a method for identifying color in an ordinary lighting environment, to thereby obviate the prior art use of providing powered light sources in the target or in an illuminating source with specific directional or color characteristics.
It would also be desirable to provide an improved method for effectively and accurately identifying color of a target image for machine and computer vision applications under varying lighting and image conditions.
It would also be desirable to provide a method for triggering interaction or applications between a user and computer by identifying colored blobs or objects of a target image in the field of view of a camera or cameras.
It would be desirable to provide a method for tracking an object such as a pen in the field of view by identifying relative location of colored sections of this object.
It would be desirable to provide a machine or computer vision system that can be used in consumer applications by identifying color.